Smart tool with integrated neural network image analysis

ABSTRACT

A smart tool includes a body having a working output. A first controller is disposed within the body and connected to a plurality of sensors. The plurality of sensors includes at least one camera having a field of view at least partially capturing the working output. A neural network is trained to analyze an image feed from the at least one camera and trained perform at least one of classifying at least one of a working tool and an extension connected to the working output, classifying a component and/or a portion of a component interfaced with the working output, and determining a positioning of the smart tool. The neural network is stored in one of the first controller and a processing unit remote from the smart tool.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Patent Application No.63/093,613 filed on Oct. 19, 2020.

TECHNICAL FIELD

The present disclosure relates generally to smart tool systems, and morespecifically to a smart tool including a neural network configured toidentify a selected tool, attachment or extension and/or a position ofthe selected tool, attachment or extension relative to a home positionor a portion of a component being worked.

BACKGROUND

Assembly systems and processes often include multiple ordered steps,with each step requiring the usage of one or more tools to install,tighten, or otherwise manipulate a portion of the assembly. In someexamples, it is important that only the correct tool or the correctamount of force be applied in a given manipulation step. In otherexamples, the steps must be performed in a certain order with a toolchange occurring in between certain steps.

Existing systems attempt to control the usage of the correct tool or thecorrect attachment for any given step by utilizing sorted tools and/orattachments that are presented to the user in a predetermined order orin predetermined locations. Such systems can be inefficient or lead toinaccurate assembly when a tool or attachment is inadvertently placed inthe wrong bin or the bins are placed in an incorrect order, or whenother human errors occur resulting in the wrong tool, attachment orextension being applied in a given step.

SUMMARY OF THE INVENTION

In one exemplary embodiment a smart tool includes a body having aworking output, a first controller disposed within the body andconnected to a plurality of sensors, the plurality of sensors includingat least one camera having a field of view at least partially capturingthe working output, and a neural network trained to analyze an imagefeed from the at least one camera and trained perform at least one ofclassifying at least one of a working tool and an extension connected tothe working output, classifying a component and/or a portion of acomponent interfaced with the working output, and determining apositioning of the smart tool, the neural network being stored in one ofthe first controller and a processing unit remote from the smart tool.

In another example of the above described smart tool the firstcontroller includes an image processing module and the first controlleris communicatively coupled to the processing unit.

In another example of any of the above described smart tools the firstcontroller is in wireless communication with the processing unit.

In another example of any of the above described smart tools the neuralnetwork is contained within the first controller.

In another example of any of the above described smart tools the atleast one camera includes a first camera fixedly connected to the body.

In another example of any of the above described smart tools the atleast one camera includes a second camera communicatively connected tothe first controller and disposed remote from the body.

In another example of any of the above described smart tools the secondcamera is a wearable camera.

In another example of any of the above described smart tools the atleast one camera includes a remote camera remote from the body andcommunicatively coupled to the controller.

In another example of any of the above described smart tools the atleast one of the working tool and the extension to the working outputincludes a visual identifier visible in the field of view, and whereinthe neural network is trained to classify the at least one of theworking tool and the extension at least partially based on the visualidentifier.

In another example of any of the above described smart tools the neuralnetwork is further configured to analyze the image feed and determinelocation of the smart tool relative to a home position.

In another example of any of the above described smart tools, the neuralnetwork is further configured to determine at least one of a physicalorientation or angle of approach of the working tool and the extensionconnected to the working output.

An exemplary method for operating a tool includes generating a least afirst video feed including at least a portion of a tool, the portionincluding a working output of the tool, identifying, using a neuralnetwork based image analysis of the at least the first video feed, atleast one of an attachment on the working output of the tool, anextension on the working output of the tool, and an element being workedby the working output of the tool, and responding to the identified atleast one of the attachment on the working output of the tool, theextension on the working output of the tool, and the element worked bythe working output of the tool by altering an operation of the tool.

In another example of the above described method for operating a toolaltering the operation of the tool includes at least one of altering aforce applied by the working output such that the force corresponds toan identified attachment and prompting a user to alter operations of thetool.

In another example of any of the above described methods for operating atool generating the at least the first video comprises generating aplurality of video feeds via cameras attached to a smart tool, andwherein identifying at least one of the attachment the extension and theelement comprises analyzing each of the video feeds in the plurality ofvideo feeds.

In another example of any of the above described methods for operating atool generating the at least the first video comprises generating asecond video feed from a wearable camera and wherein the wearable camerawirelessly communicates the second video feed to a smart toolcontroller.

In another example of any of the above described methods for operating atool the neural network image analysis is performed by a controllerlocal to the tool.

In another example of any of the above described methods for operating atool the neural network image analysis is performed by a processing unitremote from the tool and in communication with a controller disposed inthe tool.

Another example of any of the above described methods for operating atool further includes preprocessing the video feed prior to analyzingthe video feed using the neural network based analysis.

In another example of any of the above described methods for operating atool responding to the identified at least one of the attachment on theworking output of the tool, the extension on the working output of thetool, and the element worked by the working output of the tool comprisesidentifying that the tool is out of a home position.

In another example of any of the above described methods for operating atool responding to the identified at least one of the attachment on theworking output of the tool, the extension on the working output of thetool, and the element worked by the working output of the tool comprisesidentifying a lack of attachments and/or tools connected to the workingoutput and identifying that a set of tools configured to be connected tothe working output and the body are disposed in a home position andoutputting an all tools home notification.

In another example of any of the above described method responding tothe identified at least one of the attachment on the working output ofthe tool, the extension on the working output of the tool, and theelement worked by the working output of the tool includes identifying atleast one of the physical orientation or angle of approach of the tooland the extension connected to the working output of the tool, comparingthe at least one of the physical orientation or angle of approach of thetool and the extension connected to the working output of the tool to anexpected at least one of the physical orientation or angle of approachof the tool and the extension connected to the working output of thetool and outputting a notification to a user in response to the at leastone of the physical orientation or angle of approach of the tool and theextension connected to the working output of the tool varying from theexpected at least one of the physical orientation or angle of approachof the tool and the extension connected to the working output of thetool.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a high level schematic workstation for assembling acomponent.

FIG. 2 schematically illustrates a smart tool according to one example.

FIG. 3 illustrates a process for using the smart tool of FIG. 2 toenhance an assembly process.

DETAILED DESCRIPTION

FIG. 1 schematically illustrates a workstation 10 with an assembly 20placed within a work area 12 on the workstation 10. The assembly 20includes multiple fasteners 22 and connection points 24, each of whichneed to be interacted with or connected in a specific order using aspecific tool or tool attachment. While illustrated herein as a simplemechanical assembly, it is appreciated that substantially more complexmachines or intricate assemblies can benefit from the smart tool systemdisclosed herein. A set of bins 30 is positioned near the assembly 20,and includes multiple fasteners and tool attachments 32 disposed insorted bins. An assembly worker uses a tool 40 (described in greaterdetail with regards to FIG. 2) to interact with the assembly 20. Thetool 40 includes a camera 42 and a working output 44. The working output44 is configured to work either an extension or a tool, which in turnworks an attachment or fastener on the assembly 20. In one example theworking output can be a rotational output configured to rotate screws,bolts, or similar fasteners. In other examples the working output can bea linear actuation, welding gun, riveting gun, a cylinder and the like.

The camera 42 is oriented such that the working output 44 and anyportion(s) of the assembly 20 being worked on via the working output 44are within the field of view of the camera 42. The video feed generatedby the camera 42 is provided to a controller 121 (illustrated in FIG. 2)and/or a processing unit 140 (illustrated in FIG. 2). Also on theworkstation 10 is a tool home 60. The tool home 60 can be a storagecase, a designated location for placing the tool 40 while not in use, orany other home/storage position of the tool. In alternative examples thehome 60 can be located remote from the workstation 60, such as on ashelving unit, in a storage bin, or similar, and the system can functionin the same manner.

In addition to using the tool 40, portions of some assembly processesutilize hand manipulation or selection of tool attachments via theuser's hand 50. In the illustrated example, a wearable camera 52 isconnected to the controller 121 or processing unit 140, and provides animage feed, in the form of video footage, to the controller in the samemanner as the camera 42 fixed to the tool 40. In some examples, thewearable camera 52 is used instead of the attached camera 42. Inalternative examples, the wearable camera 52 can be the camera 42detached from the tool 40 and connected to a wearable strap 54. Inalternative examples, the wearable camera 52 can be worn at otherpositions, including on the forehead, on a finger, or any other relevantpositioning and can supplement the video feed from the attached camera42.

In yet further examples, the smart tool neural network analysis isincorporated into a remote analysis computer (referred to as aprocessing unit or a remote processing unit) and utilizes the wearablecamera 52 in conjunction with one or more hand tools in a similarfashion to a fully integrated smart tool.

With continued reference to FIG. 1, FIG. 2 illustrates an exemplarysmart tool 100. The smart tool 100 includes multiple cameras 110 each ofwhich has a field of view including at least a portion of a workingoutput 120 of the tool 100. The working output 120 includes a connector122 configured to connect the working output 120 to either a tool 150 oran extension, and configured to translate work from the working output120 to the tool 150 or extension. Each of the cameras 110 is connectedto a controller 121. The controller 121, in the illustrated example, isalso connected to an activation button 130 which activates one or moreoperations of the smart tool 100. In alternative examples, morecomplicated tools including multi-speed operations, multiple distinctoperations, or other features can have the operation controls affectedor controlled by the controller 121 based on the image analysis of thesmart tool system 100. In the illustrated example, the controller 121also includes a wireless communicator 124.

The wireless communication system 124 can utilize any short rangewireless communication protocol to communicate between the controller121 and a remote camera 52 (see FIG. 1), between controller 121 and aremotely located processing unit 140 storing the neural network analysissystem, or both. In alternative examples the connection can be a wiredconnection directly connecting the remote camera 52 to the processingunit 140 or controller 121, and determine one or more features of theimage.

In some examples the controller 121 includes an image processing module126. The image processing module 126 is configured to preprocess theimage feed generated by the cameras 110. The pre-processing can utilizeany image processing procedure and improves the ability to identifyimages within the processed video feed using a neural network analysis.By way of example, the image pre-processing can include any ofsharpening and/or blurring images, converting to gray scale and/or othercolor spectrum conversions, masking out regions of the image, digitalzoom or image size changes, and/or any other pre-processing orcombination of pre-processing procedures. In example systemsincorporating a detachable camera 110, the detachable camera can includethe pre-processing locally, or the pre-processing can be incorporatedinto the controller 126.

With reference to both FIG. 1 and FIG. 2, either the controller 122 orthe remotely located processing unit 140 includes a neural network. Theneural network can be a convolutional neural network, or any similarneural network, and is trained to analyze images received from thecameras 42, 52, 110, and determine one or more features in the image.

In one example the features determined include the specific tool orattachment connected to the working output. In another example, thefeatures determined include identifying a specific position on theassembly 20 (e.g. a specific bolt number) that is being worked,identifying an orientation of the tool or working output, and/oridentifying a type of element (e.g. distinguishing between boltconnection and a screw connection) being worked. In yet furtherexamples, the features determined include a combination of the above.

FIG. 3 schematically illustrates a first process 200 for using theneural network analysis of the smart tool 100 of FIG. 2 and describedgenerally above to ensure that a correct tool is attached, or thecorrect work is applied for a current operation. Initially an image feedgenerated from one or more cameras is provided to the controller 121 ina “Receive Image Feed” step 210. In examples where the neural networkanalysis is performed on the remote processing unit 140, the controller121 can perform a pre-processing function to condition the image feedfor analysis or act as a passthrough to provide the image feed to theneural network on the processing 140 depending on the configuration ofthe overall system. In other examples, the received image feed isanalyzed using a neural network disposed on the controller 121 itself.In further examples, the received image feed can include multiple imagefeds from multiple cameras disposed about the smart tool and/or wearablecameras connected to the smart tool.

The neural network uses video and image analysis to determine thepresence of an attachment, tool, or extension in the image in an“Identify Feature in Image” step 220. Once the neural network analysisdetermines that a feature is present in the image feed, a processor ineither the controller 121 or the image analysis system 140 classifiesthe identified feature to determine what type of feature is presentand/or what portion after the assembly is being worked on. In someexamples, the neural network analysis further determines a physicalorientation or angle of approach of the attachment tool or extensionduring the identify feature in image step. In one example, the neuralnetwork analysis identifies that a socket (the feature) has beenselected by the user's hand, and the neural network analyzes the socketto determine what size and/or shape socket has been selected. In anotherexample, the neural network analysis identifies that an extension (thefeature) has been attached to the working output 120 of the tool 100,and the identify feature in image step 220 determines the type ofextension that is attached. In another example, the neural networkidentifies that the tool is being used on a specific position (thefeature(s)) of the assembly, or on a specific type of attachment for theassembly. In another example, the neural network identifies that thetool is being used or held in a given orientation at a given angle.

In some examples the neural network can be configured to determine thetype of feature, tool angle, and/or tool orientation based purely ontraining the neural network to recognize image characteristics inherentto the feature. In other examples, the feature(s) can include markingsincorporated on the feature with the markings identifying one or moreparameters (e.g. size, metric/imperial, etc.) of the feature. In thisexample the features or markings do not impact the mechanicalfunctioning of the tool, attachment or extension and assist the neuralnetwork in identifying the specific features of the tool, attachment orextension once it has been identified by the neural network analysis. Inalternative examples, the markings can include physical intrusionsand/or protrusions (such as engravings or reliefs) that assist theneural network in identifying the type of features in the image.

In addition to performing the neural network analysis of the image feed,the controller 121 and/or analysis system 140 monitors a current step ofa process being performed, including a type of tool, attachment orextension that is required to be used for the current step of theprocess and/or a type of tool or attachment required for working on theidentified feature of the assembly. Once the actual tool, attachment, orextension of the feature in the image feed(s) is identified by theneural network, the controller 121 and/or the processor 140 compares thedetermined tool, attachment or extension to the expected tool,attachment or extension in a “Compare Identified Feature(s) to ExpectedFeature(s) for Step” step 230. In alternative examples, the comparisonstep 230 can correlate the type of tool, attachment or extension with atype of connection or fastener within the field of view of one or moreof the cameras 42, 52, 110 by identifying the type of connection orfastener using the image analysis.

When the identified tool, attachment, or extension matches the expectedtool, attachment or extension the process determines that the correcttool, attachment or extension for the current step is selected, or whenthe identified tool, attachment, or extension corresponds to theidentified connection or fastener, the process is allowed to continuenormal operations in an “Allow Process to Proceed” step 240. In someexamples, allowing the process to proceed includes setting a forceoutput, such as a magnitude of torque, corresponding to the selectedstep and tool, attachment or extension.

When the identified feature does not match the expected feature, or doesnot correspond to the identified connection or fastener, the systemidentifies that the incorrect tool, attachment or extension has beenselected or the smart tool is interfaced with the incorrect feature ofthe assembly and alerts the user of the incorrect selection in an “AlertUser” step 250. In some examples, when it is determined that theincorrect tool, attachment, or extension has been selected thecontroller 121 can prevent operation of the tool 100 until the correcttool, attachment or extension is selected.

In one further example, each step can include a defined comfortableorientation and/or angle of approach of the tool or working attachment.When the determined orientation or angle of attack does not match thedefined comfortable orientation and/or angle of approach of the tool orworking attachment, the alert user step can either prevent operation ofthe tool or alert the user of a more comfortable orientation or angle ofapproach.

In yet further examples, the system can adjust the type or amount ofworking output dependent on the identified feature. By way of example,when the identified feature is a specific type of fastener, is working aspecific part of the assembly, or the tool is approaching the identifiedfeature from a specific angle, the smart tool is configured to respondby applying a preset magnitude of torque corresponding to the identifiedfeature or combination of features. Similarly, when the identifiedfeature includes a specific fastener 22 or connection point 24 on theassembly that requires a particularly defined output from the workingoutput 120, the processing unit 140 determines that the identifiedfeature is the portion being worked and automatically adjusts theworking output 120 to provide the corresponding magnitude and type ofworking output. To facilitate the correct preset working outputs foreach specific connection point or fastener, the neural network istrained to recognize each connection point or fastener from multiplecamera angles and multiple operations, and the particular outputs ofeach tool type for a given connection point or fastener are correlatedwith the connection point or fastener during the training.

It is further understood that any of the above described concepts can beused alone or in combination with any or all of the other abovedescribed concepts. Although an embodiment of this invention has beendisclosed, a worker of ordinary skill in this art would recognize thatcertain modifications would come within the scope of this invention. Forthat reason, the following claims should be studied to determine thetrue scope and content of this invention.

1. A smart tool comprising: a body including a working output; a firstcontroller disposed within the body and connected to a plurality ofsensors, the plurality of sensors including at least one camera having afield of view at least partially capturing the working output; and aneural network trained to analyze an image feed from the at least onecamera and trained perform at least one of classifying at least one of aworking tool and an extension connected to the working output,classifying a component and/or a portion of a component interfaced withthe working output, and determining a positioning of the smart tool, theneural network being stored in one of the first controller and aprocessing unit remote from the smart tool.
 2. The smart tool of claim1, wherein the first controller includes an image processing module andthe first controller is communicatively coupled to the processing unit.3. The smart tool of claim 2, wherein the first controller is inwireless communication with the processing unit.
 4. The smart tool ofclaim 2, wherein the neural network is contained within the firstcontroller.
 5. The smart tool of claim 1, wherein the at least onecamera includes a first camera fixedly connected to the body.
 6. Thesmart tool of claim 5, wherein the at least one camera includes a secondcamera communicatively connected to the first controller and disposedremote from the body.
 7. The smart tool of claim 6, wherein the secondcamera is a wearable camera.
 8. The smart tool of claim 1, wherein theat least one camera includes a remote camera remote from the body andcommunicatively coupled to the controller.
 9. The smart tool of claim 1,wherein the at least one of the working tool and the extension to theworking output includes a visual identifier visible in the field ofview, and wherein the neural network is trained to classify the at leastone of the working tool and the extension at least partially based onthe visual identifier.
 10. The smart tool of claim 1, wherein the neuralnetwork is further configured to analyze the image feed and determine alocation of the smart tool relative to a home position.
 11. The smarttool of claim 1, wherein the neural network is further configured todetermine at least one of a physical orientation or angle of approach ofthe working tool and the extension connected to the working output. 12.A method for operating a tool comprising: generating a least a firstvideo feed including at least a portion of a tool, the portion includinga working output of the tool; identifying, using a neural network basedimage analysis of the at least the first video feed, at least one of anattachment on the working output of the tool, an extension on theworking output of the tool, and an element being worked by the workingoutput of the tool; and responding to the identified at least one of theattachment on the working output of the tool, the extension on theworking output of the tool, and the element worked by the working outputof the tool by altering an operation of the tool.
 13. The method ofclaim 12, wherein altering the operation of the tool includes at leastone of altering a force applied by the working output such that theforce corresponds to an identified attachment and prompting a user toalter operations of the tool.
 14. The method of claim 12, whereingenerating the at least the first video comprises generating a pluralityof video feeds via cameras attached to a smart tool, and whereinidentifying at least one of the attachment the extension and the elementcomprises analyzing each of the video feeds in the plurality of videofeeds.
 15. The method of claim 12, wherein generating the at least thefirst video comprises generating a second video feed from a wearablecamera and wherein the wearable camera wirelessly communicates thesecond video feed to a smart tool controller.
 16. The method of claim12, wherein the neural network image analysis is performed by one of acontroller local to the tool and a processing unit remote from the tooland in communication with a controller disposed in the tool.
 17. Themethod of claim 12, further comprising preprocessing the video feedprior to analyzing the video feed using the neural network basedanalysis.
 18. The method of claim 12, wherein responding to theidentified at least one of the attachment on the working output of thetool, the extension on the working output of the tool, and the elementworked by the working output of the tool comprises identifying that thetool is out of a home position.
 19. The method of claim 12, whereinresponding to the identified at least one of the attachment on theworking output of the tool, the extension on the working output of thetool, and the element worked by the working output of the tool comprisesidentifying a lack of attachments and/or tools connected to the workingoutput and identifying that a set of tools configured to be connected tothe working output and the body are disposed in a home position andoutputting an all tools home notification.
 20. The method of claim 12,wherein responding to the identified at least one of the attachment onthe working output of the tool, the extension on the working output ofthe tool, and the element worked by the working output of the toolcomprises identifying at least one of the physical orientation or angleof approach of the tool and the extension connected to the workingoutput of the tool, comparing the at least one of the physicalorientation or angle of approach of the tool and the extension connectedto the working output of the tool to an expected at least one of thephysical orientation or angle of approach of the tool and the extensionconnected to the working output of the tool and outputting anotification to a user in response to the at least one of the physicalorientation or angle of approach of the tool and the extension connectedto the working output of the tool varying from the expected at least oneof the physical orientation or angle of approach of the tool and theextension connected to the working output of the tool.